Skip to content

Conversation

@liustve
Copy link
Contributor

@liustve liustve commented May 16, 2025

Supporting ADOT auto instrumentation to handle LLO log events.

Description of changes:

  1. Adds AwsBatchLogRecordProcessor a backwards compatible custom logs BatchProcessor which has the following invariants:
    • The unserialized, uncompressed data size of exported batches will ALWAYS be <= 1 MB except for the case below:
    • If the data size of an exported batch is ever > 1 MB then the batch size is always length 1
  2. OTLPAwsLogExporter: Adds a new behavior for Retry delay based on server-side response of Retry-After header. Injects the LLO header flag if the size of the exported data > 1 MB.
  3. Customize the auto instrumentation to use the new AwsBatchLogRecordProcessor

Testing:

TODO:

  1. Add unit tests to validate behavior of AwsBatchLogRecordProcessor and OTLPAwsLogExporter
  2. E2E testing to validate any performance hits and compatibility with: BedrockAgentCore LLO Support in ADOT SDK #361

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@liustve liustve requested a review from a team as a code owner May 16, 2025 19:19
@liustve liustve marked this pull request as draft May 16, 2025 19:19
@liustve liustve requested review from mxiamxia and yiyuan-he May 16, 2025 19:20
@liustve liustve self-assigned this May 16, 2025
@liustve liustve changed the title Support for LLO in Logs Pipeline [WIP] Support for LLO in Logs Pipeline May 16, 2025
@liustve liustve changed the title [WIP] Support for LLO in Logs Pipeline [WIP] Support for handling LLO in OTLP AWS Logs Exporter May 16, 2025
## What does this pull request do?
Adds support to handle LLO from third-party instrumentation SDKs in ADOT
SDK.

The following SDKs are supported:
- Traceloop/Openllmetry
- OpenInference
- OpenLit

Note: OTel dependencies in ADOT SDK have been loosened as a short-term
workaround to support the various conflicting dependency requirements of
third-party instrumentation SDKs.

## Test plan
Built this custom ADOT SDK into various sample apps and exported the
span and logs data to the OTLP X-Ray and Logs endpoint, respectively, to
validate the LLO extraction and transformation to Gen AI Events.

Configurations tested:
- LangChain + Traceloop/Openllmetry
- LangChan + OpenInference
- LangChain + OpenLit
- CrewAI + Traceloop/Openllmetry
- CrewAI + OpenInference
- CrewAI + OpenLit

Environment variable configuration:
```
λ  env OTEL_METRICS_EXPORTER=none \
       OTEL_TRACES_EXPORTER=otlp \
       OTEL_LOGS_EXPORTER=otlp \
       OTEL_PYTHON_DISTRO=aws_distro \
       OTEL_PYTHON_CONFIGURATOR=aws_configurator \
       OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf \
       OTEL_EXPORTER_OTLP_LOGS_HEADERS="x-aws-log-group=test,x-aws-log-stream=default" \
       OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://xray.us-east-1.amazonaws.com/v1/traces \
       OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=https://logs.us-east-1.amazonaws.com/v1/logs \
       OTEL_RESOURCE_ATTRIBUTES="service.name=langchain-app" \
       OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED="true" \
       OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT="true" \
       OTEL_PYTHON_DISABLED_INSTRUMENTATIONS="http,sqlalchemy,psycopg2,pymysql,sqlite3,aiopg,asyncpg,mysql_connector,botocore,boto3,urllib3,requests,starlette" \
       AGENT_OBSERVABILITY_ENABLED="true" \
       python app.py
```

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
@liustve liustve force-pushed the sigv4_genai_logs branch from d5798db to 93aecc5 Compare May 18, 2025 00:14
@liustve liustve force-pushed the sigv4_genai_logs branch from 770f906 to b370cb0 Compare May 18, 2025 00:25
@liustve
Copy link
Contributor Author

liustve commented May 18, 2025

Closing for:
#364

@liustve liustve closed this May 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants